ERIC ED547658: Teacher Analysis of Student Knowledge (TASK): A Measure of Learning Trajectory-Oriented Formative Assessment

Q>ke

CONSORTIUM/orPOLICY/?ESEARCH/«EDUCATION

TASK

Teacher Analysis of Student Knowledge:

A Measure of Learning Trajectory-Oriented
Formative Assessment

TASK Info

Report LTO-FA Theory

Overview

M ^

TASK Instrument

Sample TASK

Domains

Results

Summary of
Findings

Jonathan Supovitz
Caroline B. Ebby
Philip Sirinides

Click on triangles to navigate to sections.

Q>ke

Home

• Report Overview
TASK Instrument
Theory of Learning Trajectory-
Oriented Formative Assessment
TASK Scoring Rubric
Uses of TASK
Sample TASK: Fractions
National Field Test
Content Knowledge
Concept Knowledge
Results

Mathematical Validity
Results

Analysis of Student Thinking
Results

Learning Trajectory: Ranking
Results

Learning Trajectory: Rationale
Results

Instructional Decision Making
Results

Summary of Findings
About CPRE

Report Overview

This interactive electronic report provides an overview of an innovative new instrument developed by
researchers at the Consortium for Policy Research in Education (CPRE) to authentically measure teachers'
formative assessment practices in mathematics. The Teacher Analysis of Student Knowledge, or
TASK, instrument assesses mathematics teachers' knowledge of formative assessment and learning
trajectories, important components of the instructional knowledge necessary to teach to the high
expectations of the Common Core State Standards (CCSS).

The electronic report has three main sections. The first section gives an overview of the TASK
instrument, the theory behind the instrument, our approach to scoring the results, and the
ways in which the instrument can be used by educators and researchers.

The second section provides a sample TASK in fractions, appropriate for teachers in
grades 3-5. This section includes the items and teacher responses in the four major
domains thattheTASK is designed to measure.

The third section provides an overview of the domains measured by the TASK
instrument and reports the results of a major national field trial conducted
by CPRE of 1,261 teachers in grades K-10 in five major mathematics
content areas: addition, subtraction, fractions, proportions, and algebra.

These results were collected from five urban and urban fringe
districts in five states in the northeast and southern United States.

They provide a sense of current teacher capacity to meet the
ambitious expectations of the Common Core State Standards in
Mathematics.

Each of these sections and its subcomponents can be
accessed either through the triangular icons on the
home page or through the table of contents on the
left.

This work was generously supported by the GE
Foundation.

©CPRE 2013

2

The TASK Instrument

Q>ke

Home

Report Overview
• TASK Instrument
Theory of Learning Trajectory-
Oriented Formative Assessment
TASK Scoring Rubric
Uses of TASK
Sample TASK: Fractions
National Field Test
Content Knowledge
Concept Knowledge
Results

Mathematical Validity
Results

Analysis of Student Thinking
Results

Learning Trajectory: Ranking
Results

Learning Trajectory: Rationale
Results

Instructional Decision Making
Results

Summary of Findings
About CPRE

A Teacher Analysis of Student Knowledge, or TASK, is a grade-specific, online assessment for mathematics
teachers which measures important components of the instructional knowledge necessary to teach to the high
expectations of the Common Core State Standards in Mathematics.

TASKS focus on the application of pedagogical content knowledge to specific student situations. They are carefully
designed to measure teachers' emphases on procedural and conceptual understanding, and their recognition of
the learning trajectories underlying core mathematics content areas. They require teachers to recognize different
levels of student understanding represented in students' work, and to explain student response strategies in
relation to research-based learning trajectories. TASKs are authentic representations of teacher understanding
because they ask teachers to respond in their own words, not select from multiple-choice options.

The TASK Instrument measures four domains in relation to
formative assessment:

1. Teachers' knowledge of mathematical concepts

2. Teachers' analysis of student understanding

3. Teachers' knowledge of mathematical learning trajectories

4. Teachers' instructional decision making

TASKS have been developed
for five mathematics
content areas:

K-1

Addition

1-2

Subtraction

3-5

Fractions

6-8

Proportion

9-10

Algebra

In addition to providing schools and districts with an understanding of teachers' preparation to teach to the
CCSS in Mathematics, TASKs can be used for professional development, program evaluation, and other research
purposes.

©CPRE 2013

3

Q>ke

Home

Report Overview
TASK Instrument
• Theory of Learning Trajectory-
Oriented Formative Assessment
TASK Scoring Rubric
Uses of TASK
Sample TASK: Fractions
National Field Test
Content Knowledge
Concept Knowledge
Results

Mathematical Validity
Results

Analysis of Student Thinking
Results

Learning Trajectory: Ranking
Results

Learning Trajectory: Rationale
Results

Instructional Decision Making
Results

Summary of Findings
About CPRE

©CPRE2013

Theory of Learning Trajectory-Oriented

Formative Assessment

The TASK instrument is based on the concept of learning trajectory-
oriented formative assessment. Formative assessment is considered
one of the most promising methods for facilitating student learning.

A synthesis of the research on formative assessment by Black & Wiliam
(1998) found substantial evidence of large gains in student learning
when teachers employ formative assessment practices. More recently,
researchers have begun to piece together the ways that students
progress as they develop mathematical understanding, called learning
trajectories (Daro, Mosher, & Corcoran, 2011).

A framework for learning trajectory-oriented formative assessment is
shown to the right. The basis of any potentially formative experience
is both a clear understanding of the gap between a learner's current
state and the goal of learning, or standard, and the pathway to achieve
the goal. A well-designed assessment helps to locate the learner on the
pathway towards the goal. The assessment becomes formative when
the information it contains provides feedback to either the learner (Feedbacki) and/or the teacher (Feedback2). For the
instructional feedback loop to close, the teacher then has to provide an informed instructional response to the learner
(Feedbacks) that helps move them closer to the goal. Knowledge of the learning trajectory helps the teacher both locate the
learner on the pathway and develop specific hypotheses about what kinds of assistance will help the learner move towards
the goal. Formative assessment is an iterative process, so the cycle repeats until the gap between the learner's current state
and the goal is closed and/or new learning goals are established.

The TASK instrument is designed to capture the teacher-related domains in this process.

Feedback (1)

Learner's

Current Assessment Learning

References

Black, R, 8c Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education: Principles, Policy & Practice, 5(1 ), 7-75.

Daro, R, Mosher, F. A., 8c Corcoran, T. (201 1 ). Learning trajectories in mathematics: A foundation for standards, curriculum, assessment, and instruction (Research Report No. 68).
Rhiladelphia, RA: Consortium for Rolicy Research in Education.

Figure excerpted from Supovitz, J. A. (2012). Getting at student understanding-The key to teachers' use of test data. Teachers College Record, 7 74(1 1 ), 1 -29.

TASK Scoring Rubric

Home

Report Overview
TASK Instrument
Theory of Learning Trajectory-
Oriented Formative Assessment
• TASK Scoring Rubric
Uses of TASK
Sample TASK: Fractions
National Field Test
Content Knowledge
Concept Knowledge
Results

Mathematical Validity
Results

Analysis of Student Thinking
Results

Learning Trajectory: Ranking
Results

Learning Trajectory: Rationale
Results

Instructional Decision Making
Results

Summary of Findings
About CPRE

Most of theTASK domains are scored on a four point rubric:

Learning Traject

Conceptual

Procedural

Teacher response draws on developmental learning trajectory to
explain student understanding or develop an instructional
response.

Teacher response focuses on underlying concepts, strategy
development, or construction of mathematical meaning.

Teacher response focuses on a particular strategy
or procedure without reference to student
conceptual understanding.

Teacher response is general or
superficially related to student work in
terms of the mathematics content.

©CPRE 2013

5

Uses of the TASK

Q-ee

Home

Report Overview
TASK Instrument
Theory of Learning Trajectory-
Oriented Formative Assessment
TASK Scoring Rubric
• Uses of TASK
Sample TASK: Fractions
National Field Test
Content Knowledge
Concept Knowledge
Results

Mathematical Validity
Results

Analysis of Student Thinking
Results

Learning Trajectory: Ranking
Results

Learning Trajectory: Rationale
Results

Instructional Decision Making

The TASK can be used in a variety of ways, including:

» Program evaluation

» Educational research on data use, mathematics pedagogical
content knowledge, or formative assessment

» Professional development for teachers of mathematics

» Assist districts to diagnose areas of strength and need in
mathematics teaching and formative assessment practices
for resource allocation, coaching, and design of professional
development

CPRE is available to provide both the infrastructure and suppi
forTASK administration and scoring. On a per-teacher basis, v
can administer the TASK at the appropriate grade level, track
completion of the instrument, score completed TASKs, or
provide training for local scorers.

For information on these or other more customize
services contact cpretask@gse.upenn.edu.

Results

Summary of Findings
About CPRE

Q>ke

Home

Report Overview

TASK Instrument

Theory of Learning Trajectory-

Oriented Formative Assessment

TASK Scoring Rubric

Uses of TASK

• Sample TASK: Fractions
National Field Test
Content Knowledge
Concept Knowledge
Results

Mathematical Validity
Results

Analysis of Student Thinking
Results

Learning Trajectory: Ranking
Results

Learning Trajectory: Rationale
Results

Instructional Decision Making
Results

Summary of Findings
About CPRE

Sample TASK: Fractions

ATASK provides teachers with a grade-appropriate problem and a set of student responses, and asks teachers to
complete seven steps:

1 . Examine the mathematics problem and state the correct answer.

2. Explain what a student at your grade level needs to know and/or understand in order to solve the problem.

3. Examine the solutions of a set of typical students and determine if their solution processes are mathematically valid.

4. Comment on four students' solution processes in terms of what the work suggests about their understanding of number and
operations (or algebraic reasoning).

5. Rank each student's solution in order of the level of sophistication of the mathematical thinking that is represented.

6. Explain the rationale for the rankings given to each student.

7. Suggest instructional next steps and explain the rationale for those steps for a student who has a correct, but less-sophisticated
response to the problem, and a student who demonstrates conceptual weakness in the response.

Item Prompt : Each carton holds 24 oranges. Kate's carton is 1/3 full. Paul's carton is 2/4 full. If they put all their oranges
together, would Kate and Paul fill 1 whole carton? Solve the problem. Show your work.

Student Responses :

Click on the buttons below to see sample teacher responses and their rubric score for each domain:

f

Concept

C \

Analysis of

A

r \

Learning Trajectory:

Instructional

\

Knowledge

J

^ Student Thinking

J

^ Rationale ^

V

Decision Making ^

©CPRE 2013

7

Q>ke

Home

Report Overview
TASK Instrument
Theory of Learning Trajectory-
Oriented Formative Assessment
TASK Scoring Rubric
Uses of TASK
Sample TASK: Fractions
• National Field Test
Content Knowledge
Concept Knowledge
Results

Mathematical Validity
Results

Analysis of Student Thinking
Results

Learning Trajectory: Ranking
Results

Learning Trajectory: Rationale
Results

Instructional Decision Making
Results

Summary of Findings
About CPRE

National Field Test

The TASK instrument was developed and piloted beginning in 2010. In Spring 2012, based on pilot results,
CPRE conducted a large field trial in partnership with five public school districts in five northeastern and
southern states. The districts varied in size, student demographics, and programs of mathematics instruction.
The table below presents the number of schools and the average number of students per grade, as well as
student demographics in each of the five districts.

1

District A

District B

District C

District D

District E

District Size^

# Schools

23

57

133

184

20

# Students

12,324

32,251

93,951

79,130

15,281

Student Demographics

% White

47%

25%

53%

14%

39%

% Economically Disadvantaged

54%

73%

63%

82%

49%

% Limited English Proficient

8%

4%

6%

11%

14%

% Special Education

18%

20%

13%

20%

9%

District Teachers

Sample Size^

325

376

394

438

318

#Teacher Respondents

273

274

268

329

242

TASK Response Rate

84%

73%

68%

75%

76%

Notes: ® Number of schools and average student per grade based on 2011 . Random sample within district stratified by grade intervals.

To achieve our final sample, we randomly drew 1 ,851 teachers from the five districts. Of these, 1 ,386
responded, for a 75% response rate. Of the completed TASKs, 42 were removed due to substantial missing
data. The data reported here also do not include the 83 responses on the geometry TASK. The final sample
we report on here consists of 1 ,261 responses for teachers in grades K-1 0. More information on the technical
qualities of the TASK can be found at cpre.org/task.

©CPRE 2013

8

Q>ke

Home

Report Overview
TASK Instrument
Theory of Learning Trajectory-
Oriented Formative Assessment
TASK Scoring Rubric
Uses of TASK
Sample TASK: Fractions
National Field Test
• Content Knowledge
Concept Knowledge
Results

Mathematical Validity
Results

Analysis of Student Thinking
Results

Learning Trajectory: Ranking
Results

Learning Trajectory: Rationale
Results

Instructional Decision Making
Results

Summary of Findings
About CPRE

Content Knowledge

The first question of the TASK asked teachers to determine the correct answer to the mathematical problem
before they were shown any student responses. Incorrect responses may provide an indication of the
teacher's weakness in content knowledge. However, since this was the only item that addressed content
knowledge we do not consider it to be a sufficient measure of teacher's content knowledge. Note: Content
knowledge is note TASK domain.

Addition (K-1)

n=246

Subtraction (1-2)

n=185

Fractions (3-5)

n=376

Proportion (6-8)

n=291

Algebra (9-10)

n=163

Correct

Incorrect

Take-Aways:

» Teachers' ability to correctly solve the given mathematical problem at their grade level was generally
strong.

» Surprisingly, 1 5%, or 24, of the 1 63 grade 9-1 0 teachers gave an incorrect answer to the algebra
problem, which involved extending and generalizing a pattern.

©CPRE 2013

9

Domain | Concept Knowledge

Home

Report Overview
TASK Instrument
Theory of Learning Trajectory-
Oriented Formative Assessment
TASK Scoring Rubric
Uses of TASK
Sample TASK: Fractions
National Field Test
Content Knowledge
• Concept Knowledge
Results

Mathematical Validity
Results

Analysis of Student Thinking
Results

Learning Trajectory: Ranking
Results

Learning Trajectory: Rationale
Results

Instructional Decision Making
Results

Summary of Findings
About CPRE

The Concept Knowledge domain measures teachers' ability to identify and articulate the mathematics concept
and related sub-concepts that are represented in a particular item. In formative assessment, it is important
for teachers to have knowledge of the type of evidence likely to be elicited by a specific problem. To assess
this domain, teachers were asked to describe what a student at their grade level would need to know and/or
understand in order to solve the given problem. Their responses were scored with the four level TASK rubric.

Q>ke

Home

Report Overview
TASK Instrument
Theory of Learning Trajectory-
Oriented Formative Assessment
TASK Scoring Rubric
Uses of TASK
Sample TASK: Fractions
National Field Test
Content Knowledge
Concept Knowledge
• Results
Mathematical Validity
Results

Analysis of Student Thinking
Results

Learning Trajectory: Ranking
Results

Learning Trajectory: Rationale
Results

Instructional Decision Making
Results

Summary of Findings
About CPRE

ts I Concept Knowledge

Addition (K-1)

n=246

Subtraction (1-2)

n=185

Fractions (3-5)

n=376

Proportion (6-8)

n=291

Algebra (9-10)

n=163

11%

49%

32%

8%

10%

46%

35%

8%

19%

31%

28%

16%

1 %

6 %

26%

34%

17%

21%

Scored using TASK rubric

NOTE: The Conceptual Response
category was further divided into
responses that had a general
conceptual focus (shown on the
graph in light blue) and responses
where concepts were more fully
articulated (shown in darker blue).
Responses were considered to be
in the learning trajectory category
(shown in green) if they included
more than one articulated concept.

2 %

ToSampleTeacher
Responses

Take-Aways:

» In grades K-5, over 40% of teacher responses reflected some degree of conceptual focus in the analysis of
the problem.Of those, a much smaller proportion of responses were fully articulated or referenced more
than one articulated underlying concept (9% in grades K-2 and 22% in grades 3-5).

» In grades K-2, nearly half of the responses focused on procedures for solving addition or subtraction
problems ratherthan underlying concepts.

» In grades 6-8, only 14% of respondents referenced concepts associated with the proportional reasoning
item, while almost half (48%) of the respondents focused on procedures; 37% referenced only general
topics, such as "ratios" or "proportions."

» In grades 9-1 0, 40% of the responses reflected some conceptual focus, while 34% focused on procedures.
About a quarter (26%) of teachers' responses referenced only a general topic, such as "algebra" or
"patterns."

11

©CPRE 2013

Sample TASK

edae

Home

Report Overview
TASK Instrument
Theory of Learning Trajectory-
Oriented Formative Assessment
TASK Scoring Rubric
Uses of TASK
Sample TASK: Fractions
National Field Test
Content Knowledge
Concept Knowledge
Results

Mathematical Validity
Results

Analysis of Student Thinking
Results

Learning Trajectory: Ranking
Results

Learning Trajectory: Rationale
Results

Instructional Decision Making
Results

Summary of Findings
About CPRE

TASK Prompt: What does a student need to know and understand in order to solve this problem?

Learning Trajectory Response

"They need to understand: A fraction is a part of a whole. A whole can be a group of things or one thing.

24 oranges is a whole, which is mentioned in this problem. 1 2/1 2 is a whole. When adding fractions
you don't add the denominator. Either how to find 1/3 and 2/4 of 24 or how to make a common
^ denominator."

Conceptual Response

"They need to know that 2/4 = 1/2 they also need to know the relationship of fourths and thirds. ..which
is bigger." (articulated)

"Understand fractions, part of a whole and know how to add." (general)

V J

Procedural Response

"They have to reduce fractions and be able to find common denominators and then add fractions."

V j

General Response

"Fractions. Reduction. Multiplication. Number Order. Ability to countto 20. Addition. English
comprehension."

To Sample TASK

r

To Domain Results

J V,

J

©CPRE 2013

12

Q>ke

Home

Report Overview
TASK Instrument
Theory of Learning Trajectory-
Oriented Formative Assessment
TASK Scoring Rubric
Uses of TASK
Sample TASK: Fractions
National Field Test
Content Knowledge
Concept Knowledge
Results

• Mathematical Validity
Results

Analysis of Student Thinking
Results

Learning Trajectory: Ranking
Results

Learning Trajectory: Rationale
Results

Instructional Decision Making
Results

Summary of Findings
About CPRE

Domain | Mathematical Validity

Mathematical Validity is a measure of teachers' ability to analyze a student's solution strategy and determine
whether the approach is mathematically sound, regardless of the final answer. In formative assessment,
teachers need to be able to look beyond the correctness of a student's answer to assess the appropriateness
of the solution method. For example, students may use an incorrect approach but still arrive at a correct
answer. In each TASK, teachers were asked to determine whether each student solution process was
mathematically valid.

In all TASKS, there were two mathematically incorrect solutions and one solution was somewhat ambiguous
in that the student attempted to use an appropriate strategy, but either made a conceptual or computational
error in the process. To score this question, we looked at the number of correctly identified responses out of
the number that were unambiguous (between 3 and 5 depending on the TASK).

Q>ke

Home

Report Overview
TASK Instrument
Theory of Learning Trajectory-
Oriented Formative Assessment
TASK Scoring Rubric
Uses of TASK
Sample TASK: Fractions
National Field Test
Content Knowledge
Concept Knowledge
Results

Mathematical Validity
• Results

Analysis of Student Thinking
Results

Learning Trajectory: Ranking
Results

Learning Trajectory: Rationale
Results

Instructional Decision Making
Results

Summary of Findings
About CPRE

Results I Mathematical Validity

The graph shows the proportion of teachers who (1 ) correctly identified all strategies (2) misidentified one
strategy or (3) misidentified two or more strategies.

Addition (K-1)

Out of 5

9% 32%

Algebra (9-10)

Out of 3

1 7% 33%

51%

Two or more incorrect
One incorrect

All correct

Take-Aways:

» More than half of teachers at all grade levels were able to correctly identify the validity of all student
strategies that were non-ambiguous.

» Fewer than a fifth of the teachers at each grade level made multiple misjudgments about the validity of
students' solution strategies.

» Taken together, these results suggest that most teachers were able to determine the mathematical
validity of a range of student solution strategies.

©CPRE 2013

14

Q>ke

Home

Report Overview
TASK Instrument
Theory of Learning Trajectory-
Oriented Formative Assessment
TASK Scoring Rubric
Uses of TASK
Sample TASK: Fractions
National Field Test
Content Knowledge
Concept Knowledge
Results

Mathematical Validity
Results

• Analysis of Student Thinking
Results

Learning Trajectory: Ranking
Results

Learning Trajectory: Rationale
Results

Instructional Decision Making
Results

Summary of Findings
About CPRE

Domain | Analysis of Student Thinking

Analysis of Student Thinking is a measure of teachers' ability to identify the underlying conceptual
understandings or misconceptions present in student responses on assessments. During formative
assessment, the interpretation of student thinking is a key precursor to a teacher's ability to provide targeted
feedback to move the learner forward. Teachers must be able to look beyond whether students produce a
correct answer or use correct procedures to also draw inferences about what the student understands and
whether there are underlying misconceptions that need to be addressed.

On the TASK, teachers were asked to comment on four students' solution processes in terms of what the work
suggested about each student's understanding of number and operations (or algebraic reasoning at the
secondary level). Each response was scored with the four-point TASK rubric to judge the extent to which the
response focused on underlying conceptual understandings that were represented in the student solution
strategies.

GOTO RESULTS

s I Analysis of Student Thinking

Home

Report Overview
TASK Instrument
Theory of Learning Trajectory-
Oriented Formative Assessment
TASK Scoring Rubric
Uses of TASK
Sample TASK: Fractions
National Field Test
Content Knowledge
Concept Knowledge
Results

Mathematical Validity
Results

Analysis of Student Thinking
• Results

Learning Trajectory: Ranking
Results

Learning Trajectory: Rationale
Results

Instructional Decision Making
Results

Summary of Findings
About CPRE

Addition (K-1)

n=246

Subtraction (1-2)

n=185

Fractions (3-5)

n=376

3%

1 %

Scored using TASK rubric

Proportion (6-8)

n=291

Algebra (9-10)

n=163

3%

ToSampleTeacher
Responses

J

Take-Aways:

» Across all grade levels, the vast majority of teacher responses were procedural, focusing on what the
student did to solve the problem rather than commenting on the student's underlying conceptual
understanding.

» Teacher responses for fractions had the greatest percentage of conceptual responses (1 8%) and a few
responses at the highest learning trajectory level (1 %). This may be a reflection of the fact that fractions is
a topic that is typically taught with a conceptual focus in the elementary grades.

» All teacher responses for proportions were either procedural or general. The large percentage of
procedural responses (79%) suggests a procedural emphasis in middle school mathematics teaching.
Additionally, about one fifth of the teachers at this grade level gave only general analyses of the student
work (e.g., "understands proportions" or "demonstrates strong reasoning").

©CPRE 2013

16

Q>re

SampleTASK | Analysis of StudentThinking

Home

Report Overview
TASK Instrument
Theory of Learning Trajectory-
Oriented Formative Assessment
TASK Scoring Rubric
Uses of TASK
SampleTASK: Fractions
National Field Test
Content Knowledge
Concept Knowledge
Results

Mathematical Validity
Results

Analysis of StudentThinking
Results

Learning Trajectory: Ranking
Results

Learning Trajectory: Rationale
Results

Instructional Decision Making
Results

Summary of Findings
About CPRE

General Response

TASK Prompt: Comment on each student's solution process in terms of what the work suggests about the
student's understanding of numbers and operations.

Learning Trajectory Response

"Abby understands that the size of fractions is determined by the denominator, and they represent
breaking the whole into equal parts. She also understands equivalent fractions as well. Thereby, she is
able to compare the two fractions and ultimately, compare her results to one whole."

Conceptual Response

"She shows that she understands the concepts of fractional partof a whole" (articulated)

"Abby understands how fractions make a whole part." (general)

Procedural Response

"Abby drew 2 pies and was able to figure out that the 2 different fractions didn't equal a whole
together."

r i

"She has a basic understanding effractions."

V

J

©CPRE 2013

17

Q>ke

Home

Report Overview
TASK Instrument
Theory of Learning Trajectory-
Oriented Formative Assessment
TASK Scoring Rubric
Uses of TASK
Sample TASK: Fractions
National Field Test
Content Knowledge
Concept Knowledge
Results

Mathematical Validity
Results

Analysis of Student Thinking
Results

• Learning Trajectory: Ranking
Results

Learning Trajectory: Rationale
Results

Instructional Decision Making
Results

Summary of Findings
About CPRE

Domain | Learning Trajectory Orientation: Ranking

This component of the TASK measures a teacher's ability to position student solution strategies along a
learning trajectory and order them in terms of the sophistication of thinking and reasoning. Learning
trajectories, an important underlying component of the Common Core State Standards in Mathematics,
provide a framework for how student thinking becomes more sophisticated and efficient over time so that
feedback can be tailored to move students forward along the path to achieving the desired goal.

On the TASK, teachers were asked to rank the student responses in order of their level of sophistication. While
there was not one correct way to precisely rank the order of responses, the student solutions could be cleanly
placed into three categories: (1 ) the student response contained evidence of solid numerical, fractional,
proportional, or algebraic reasoning; (2) the student response had evidence of transitional thinking in
numerical, fractional, proportional, or algebraic reasoning; (3) the student response had no evidence of
numerical, fractional, proportional, or algebraic reasoning.

If the teacher correctly ordered the student responses in relation to these categories, then the response was
considered to be ordered correctly.

For this question the four rubric levels were defined to represent the degree to which the ranking reflected
attention to student reasoning.

Click on the icon for
more information on
the Learning Trajectory
Orientation: Ranking
scoring rubric.

GOTO RESULTS

W

©CPRE 2013

18

.earning

Home

Report Overview
TASK Instrument
Theory of Learning Trajectory-
Oriented Formative Assessment
TASK Scoring Rubric
Uses of TASK
Sample TASK: Fractions
National Field Test
Content Knowledge
Concept Knowledge
Results

Mathematical Validity
Results

Analysis of Student Thinking
Results

Learning Trajectory: Ranking
Results

Learning Trajectory: Rationale
Results

Instructional Decision Making
Results

Summary of Findings
About CPRE

Score

4

3

2

©CPRE 2013

ra ector

ation:

<ing I Scoring Rubric

Ranking Explanation

Correct order and most
sophisticated thinking
identified

Advanced learning
trajectory orientation

Correct order

Evidence of learning
trajectory orientation

Incorrect order but the lowest
two responses were in the
bottom two-thirds of the
ranking

Ability to identify correct and
incorrect reasoning

Incorrect order and one of
the lowest two responses
was ranked in the top two

No emphasis on Student
reasoning or prioritizing use of
specific method over conceptual
or procedural understanding

GOTO RESULTS ^

19

Q>ke

Home

Report Overview
TASK Instrument
Theory of Learning Trajectory-
Oriented Formative Assessment
TASK Scoring Rubric
Uses of TASK
Sample TASK: Fractions
National Field Test
Content Knowledge
Concept Knowledge
Results

Mathematical Validity
Results

Analysis of Student Thinking
Results

Learning Trajectory: Ranking
• Results

Learning Trajectory: Rationale
Results

Instructional Decision Making
Results

Summary of Findings
About CPRE

s I Learning Trajectory Orientation: Ranking

Addition (K-1)

n=246

Subtraction (1-2)

n=185

Fractions (3-5)

n=376

Proportion (6-8)

n=291

Algebra (9-10)

n=163

15%

52%

CNI

CNI

10%

13%

7%

27%

r 54%

Scored using LTO rubric.
Click on icon for more info.

Take-Aways:

» In grades K-2, for both addition and subtraction, the distribution of rubric scores followed a similar pattern, with more than
a third of the respondents able to correctly order student responses in relation to the sophistication of reasoning.

» In grades 3-5, while there were fewer responses reflecting an advanced orientation, a third of the responses reflected the
correct general order (shown in blue and green).

» In grades 6-8, only 14% of teachers selected the correct order in relation to evidence of proportional reasoning and nearly
a quarter (24%) ranked one of the lowest two responses (those containing minimal evidence of proportional reasoning) as
being advanced. Six out of 1 0 teachers had some, although imperfect, sense of appropriate rankings.

» In grades 9-10, by contrast, 81% of teachers correctly ordered the student strategies and more than half of the teachers
were able to order them in relation to the sophistication of reasoning. The higher scores on this TASK may be due to a
stronger sense of learning trajectories, or alternatively to the fact that only four pieces of student work were presented
(compared to the six pieces presented in the otherTASKs).

» Overall, less than half of teachers in grades K-8 correctly ordered the student strategies in relation to student reasoning.

©CPRE 2013

20

Q>ke

Home

Report Overview
TASK Instrument
Theory of Learning Trajectory-
Oriented Formative Assessment
TASK Scoring Rubric
Uses of TASK
Sample TASK: Fractions
National Field Test
Content Knowledge
Concept Knowledge
Results

Mathematical Validity
Results

Analysis of Student Thinking
Results

Learning Trajectory: Ranking
Results

• Learning Trajectory: Rationale
Results

Instructional Decision Making
Results

Summary of Findings
About CPRE

Domain | Learning Trajectory Orientation: Rationale

In addition to the rankings, teachers were also asked to provide their rationales for their rankings of student
work. These responses were scored based upon each teacher's focus when ranking the solution strategies
(general, procedural, conceptual, or learning trajectory). Rationales were scored in two stages: first, the rater
examined the explanations provided by the teacher for the student solutions they ranked in the top three
to determine what the teacher was attending to when evaluating successful student work. Second, the rater
examined the rationales for the student solutions ranked in the bottom three to see how the teacher was
evaluating weaknesses in student work. The result graphs show the average of these two scores.

Q>ke

Home

Report Overview
TASK Instrument
Theory of Learning Trajectory-
Oriented Formative Assessment
TASK Scoring Rubric
Uses of TASK
Sample TASK: Fractions
National Field Test
Content Knowledge
Concept Knowledge
Results

Mathematical Validity
Results

Analysis of Student Thinking
Results

Learning Trajectory: Ranking
Results

Learning Trajectory: Rationale
• Results

Instructional Decision Making
Results

Summary of Findings
About CPRE

©CPRE2013

s I Learning Trajectory Orientation: Rationale

Addition (K-1)

n=246

Subtraction (1-2)

n=185

Fractions (3-5)

n=376

Proportion (6-8)

n=291

Algebra (9-10)

n=163

Take-Aways:

» Across grade levels, the vast majority of teachers explained their ranking rationales by pointing out
procedural aspects of students' work rather than what students understood or how that understanding
was situated in a learning trajectory.

» Comparing these results to the Learning Trajectory Orientation Rankings, we see that teachers were
more successful in choosing the correct ranking than they were in providing a reasoned rationale for
that ranking. This suggests that while many teachers may have a sense for students' sophistication of
reasoning, this knowledge is not well articulated in relation to the developmental nature of student
thinking.

» The relatively high percentage (28%) of grade 9-1 0 algebra teachers who gave conceptual or learning
trajectory-oriented explanations to explain their rankings lends some credence to the earlier mentioned
hypotheses that high school teachers, with more subject-specific knowledge, may be better able to
recognize and articulate student conceptual understanding. 22

5%

Scored using TASK rubric

1 %

1 %

15 %

57%

26%

2% [To Sample Teacher
Responses

J

Q>ke

Home

Report Overview
TASK Instrument
Theory of Learning Trajectory-
Oriented Formative Assessment
TASK Scoring Rubric
Uses of TASK
Sample TASK: Fractions
National Field Test
Content Knowledge
Concept Knowledge
Results

Mathematical Validity
Results

Analysis of Student Thinking
Results

Learning Trajectory: Ranking
Results

Learning Trajectory: Rationale
Results

Instructional Decision Making
Results

Summary of Findings
About CPRE

SampleTASK| Learning Traj’ector

tation: Kaiiona e

TASK Prompt: Explain your ranking of each student's response in relation to the responses of the other
students.

Learning Trajectory Response

"I ranked Abby second because she uses a basic model while Emma needs reasoning alone to solve the
problem. This shows a deeper conceptual understanding. Although Devon's answer was incorrect, his
reasoning was sound. "

V J

Conceptual Response

"Brad understand equivalent fractions, and the need for common denominators in order to add
fractions. Emma understands that what fractional pieces represent. Abby has a concrete and correct
understanding of equal pieces (fractions) represent the whole. She understands equivalency of a basic
^ fraction. ..1/2=2/4." ^

Procedural Response

"Brad showed that he understands how to properly add fractions with different denominators. Abby I
used pictures to accurately show representations and got her answer by combining them. Devon
attempted to use pictures but miscounted which led to the wrong answer."

General Response

"Brad converted the fractions to 12ths. Emma explained completely with words. Abby explained with
words and pictures."

V J

r \

r \

ToSampleTASK

To Domain Results

V j

J

©CPRE 2013

23

Q>ke

Home

Report Overview
TASK Instrument
Theory of Learning Trajectory-
Oriented Formative Assessment
TASK Scoring Rubric
Uses of TASK
Sample TASK: Fractions
National Field Test
Content Knowledge
Concept Knowledge
Results

Mathematical Validity
Results

Analysis of Student Thinking
Results

Learning Trajectory: Ranking
Results

Learning Trajectory: Rationale
Results

• Instructional Decision Making
Results

Summary of Findings
About CPRE

Domain | Instructional Decision Making

The final domain in the TASK measures teachers' ability to choose an appropriate instructional response to
move students from their current level of understanding along the developmental trajectory towards greater
understanding. In formative assessment, feedback to the learner can be more effective when it is specifically
tailored to students' developmental levels.

On the TASK, teachers were asked to provide next steps, and explain their rationale for those steps, for two
students: one student who had a correct, but less-sophisticated response to the problem, and a second
student who demonstrated conceptual weakness in their response. The rubric reflects the four levels of
teacher response (general, procedural, conceptual, and learning trajectory). In order for a response to be
considered at the highest level (learning trajectory-oriented) it could be procedural or conceptual, but had
to build on the current student understanding to either move the student incrementally towards a more
sophisticated strategy or solidify the current strategy by addressing nnisconceptions.

GOTO RESULTS

©CPRE 2013

24

s I Instructional Decision Making

Home

Report Overview
TASK Instrument
Theory of Learning Trajectory-
Oriented Formative Assessment
TASK Scoring Rubric
Uses of TASK
Sample TASK: Fractions
National Field Test
Content Knowledge
Concept Knowledge
Results

Mathematical Validity
Results

Analysis of Student Thinking
Results

Learning Trajectory: Ranking
Results

Learning Trajectory: Rationale
Results

Instructional Decision Making
• Results
Summary of Findings
About CPRE

Addition (K-1)

n=246

Subtraction (1-2)

n=185

Fractions (3-5)

n=376

Proportion (6-8)

n=291

Algebra (9-10)

n=163

Take-Aways:

» Across all grade levels, the majority of teachers' suggestions to students focused on procedural advice
(i.e., on teaching a student a particular strategy or procedure), rather than on developing mathematical
meaning or understanding.

» In grades K-1, more than three quarters (77%) of the teachers gave procedural suggestions and fewer
than 1 0% of the instructional responses sought to deepen students' conceptual understanding.

» By contrast, while procedural responses still predominated (nearly 60%) in subtraction, fractions, and
proportions, about 20% of teachers were able to make conceptual suggestions.

» Teachers in grades 9-10 provided the strongest responses. Nearly a third gave conceptual responses
with an additional 8% reflecting a learning trajectory orientation (i.e., building on current student
understanding to address misconceptions or help the student develop a more sophisticated strategy).

19 %

59%

20 %

2 %

Scored using TASK rubric

22 %

60%

16 %

2 %

28%

55%

14 %

3%

15%

46%

30%

9%

ToSampleTeacher

Responses

©CPRE 2013

25

Q>ke

Home

Report Overview
TASK Instrument
Theory of Learning Trajectory-
Oriented Formative Assessment
TASK Scoring Rubric
Uses of TASK
Sample TASK: Fractions
National Field Test
Content Knowledge
Concept Knowledge
Results

Mathematical Validity
Results

Analysis of Student Thinking
Results

Learning Trajectory: Ranking
Results

Learning Trajectory: Rationale
Results

Instructional Decision Making
Results

Summary of Findings
About CPRE

Sample TASK | Instructiona

akina

TASK Prompt: Examine Abby's work. As a teacher, what would you do next? Please explain your rationale for the
steps you suggest.

Learning Trajectory Response

"First, I would encourage herto draw a picture and group things according to the fraction. ..1/2 = 12/24, 1/3 = 8/24.

I would explore the relationship among those equivalencies to help her understand the interrelatedness. Drawing
a picture is a basic understanding or step to equal - sharing/division - fractions but easy for children to do at an early
age. Equivalent fractions is more sophisticated but can be explored to understand how these numbers make sense."

Conceptual Response

"I would first ask Ab by to look at her representation of 1/3 and ask herto explain how it is indeed 1/3.Abby needs to
understand that the circle must be divided into 3 EQUAL parts. Next, I would ask how can she prove it does not make
one whole when it is added to one half? I would guide her in seeing that 1/3 (a whole divided into 3 equal parts) is
^ less than one whole divided into 2 equal parts and therefore, when added to 1/2 it could not equal one whole."

Procedural Response

"Abby would be directed into writing fractions, determining a common denominator and then making
equivalent fractions and solving the problem."

V y

General Response

^ 1

"Abby should practice to enrich her understanding."

V

J

To SampleTASK

^ r

To Domain Results

©CPRE 2013

V.

26

Q>ke

Home

Report Overview
TASK Instrument
Theory of Learning Trajectory-
Oriented Formative Assessment
TASK Scoring Rubric
Uses of TASK
Sample TASK: Fractions
National Field Test
Content Knowledge
Concept Knowledge
Results

Mathematical Validity
Results

Analysis of Student Thinking
Results

Learning Trajectory: Ranking
Results

Learning Trajectory: Rationale
Results

Instructional Decision Making
Results

• Summary of Findings
About CPRE

Summary of Findings

These findings reflect a general picture of the current state of teachers' learning trajectory-oriented formative
assessment capabilities in grades K-10 in five urban and urban fringe districts in five states. Overall, they
indicate:

» Across the domains examined on the TASK, there were more procedural responses than any other category. In fact,
with the exception of Analysis of Student Thinking in fractions, the procedural responses outnumbered conceptual
and learning trajectory responses combined. Given the emphasis in the Common Core State Standards on rigor as
a balance between conceptual and procedural understanding, this suggests that there is a great deal of room for
growth in teacher capacity to identify, interpret, and respond to students' conceptual understanding .

» Although about 40% of teachers are able to identify the mathematics concept that an item intends to assess,
when examining the student work of that item, more than three quarters of the teachers focused on what
students do (procedural) rather than what they understand (conceptual) .

» General and procedural interpretations were most predominant in the responses from middle school (grades 6-8)
teachers on the proportions TASK, suggesting that this is a particular area of need for professional development
focusing on conceptual understanding.

» After examining specific pieces of student work, the majority of teachers across all grade levels suggested
teaching the student particular strategies or procedures rather than developing mathematical meaning or
understanding .

» Teachers were more successful at ranking student work in order of sophistication than they were in providing a
reasoned rationale for that ranking . This suggest that while many teachers may have developed a tacit sense of levels
of sophistication in student reasoning, their knowledge about the developmental nature of student thinking is not
well articulated.

» While there is much room for further development, teachers in grades 9-10 (algebra) performed stronger on the TASK
than other teachers . This could be due to the fact that they are more subject matter specialized than elementary school
teachers or that they have more experience analyzing different strategies in student work.

©CPRE 2013

27

Q-ee

Home

Report Overview
TASK Instrument
Theory of Learning Trajectory-
Oriented Formative Assessment
TASK Scoring Rubric
Uses of TASK
Sample TASK: Fractions
National Field Test
Content Knowledge
Concept Knowledge
Results

Mathematical Validity
Results

Analysis of Student Thinking
Results

Learning Trajectory: Ranking
Results

Learning Trajectory: Rationale
Results

Instructional Decision Making
Results

Summary of Findings
• About CPRE

CON$ORTIUM/orPOLICY/?E$EARCHmEDUCATION

The Consortium for Policy Research in Education (CPRE) brings together education
experts from renowned research institutions to contribute new knowledge that informs
K-12 education policy and practice. Our work is peer-reviewed and open-access for
education policymakers, practitioners, and researchers at cpre.org.

University of Pennsylvania | Teachers College, Columbia University | Harvard University | Stanford University
University of Michigan | University of Wisconsin-Madison | Northwestern University

©CPRE 2013

28